{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "52d3a943",
   "metadata": {},
   "source": [
    "<img src='http://www-scf.usc.edu/~ghasemig/images/sharif.png' alt=\"SUT logo\" width=200 height=200 align=left class=\"saturate\" >\n",
    "\n",
    "<br>\n",
    "<font face=\"Times New Roman\">\n",
    "<div dir=ltr align=center>\n",
    "<font color=0F5298 size=7>\n",
    "    Introduction to Machine Learning <br>\n",
    "<font color=2565AE size=5>\n",
    "    Computer Engineering Department <br>\n",
    "    Fall 2022<br>\n",
    "<font color=3C99D size=5>\n",
    "    Homework 3: Practical - ML Models for Tabular Datasets <br>\n",
    "<font color=696880 size=4>\n",
    "    Niloufar Razani \n",
    "    \n",
    "    \n",
    "____"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afcf1f24",
   "metadata": {},
   "source": [
    "### Full Name : \n",
    "### Student Number : \n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c097381",
   "metadata": {},
   "source": [
    "<font face=\"Times New Roman\" size=4><div dir=ltr>\n",
    "In this homework we are going to implement Adaboost algorithm from scratch. Please read this chapter's <a href=\"https://github.com/asharifiz/Introduction_to_Machine_Learning/tree/main/Jupyter_Notebooks/Chapter_04_Tabular_Data_Models\"><font face=\"Roboto\">notebook</font></a> and then complete the #TODO sections. <br>\n",
    "We will use the heart_disease.csv dataset, which you can see more details about in this <a href=\"https://www.kaggle.com/datasets/johnsmith88/heart-disease-dataset?resource=download&select=heart.csv\"><font face=\"Roboto\">Link</font></a>.\n",
    "___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a687a71",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b152c1e",
   "metadata": {},
   "source": [
    "### Data Prepration (10 points) \n",
    "1.   Load Dataset\n",
    "2.   Separate target feaure\n",
    "3.   Change class labels to 1 and -1\n",
    "4.   Do train-test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "159d3986",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "695a1fc7",
   "metadata": {},
   "source": [
    "### Adaboost Algorithm Implementation (40 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95f76156",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "def calculate_error(y, y_pred, w):\n",
    "    #TODO : Calculate the weighted error of a weak classifier.\n",
    "    \n",
    "\n",
    "def calculate_alpha(error):\n",
    "    #TODO : Calculate the weight of a weak classifier.\n",
    "    \n",
    "\n",
    "def update_weights(w, alpha, y, y_pred):\n",
    "    #TODO : Update weights after a boosting iteration.\n",
    "    \n",
    "\n",
    "    \n",
    "class AdaBoost:\n",
    "    \n",
    "    def __init__(self):\n",
    "        self.alphas = []\n",
    "        self.G_M = []\n",
    "        self.training_errors = []\n",
    "\n",
    "    def fit(self, X, y, M = 100):\n",
    "        \n",
    "        self.alphas = [] \n",
    "        self.training_errors = []\n",
    "        self.M = M\n",
    "\n",
    "        for m in range(0, M):\n",
    "            \n",
    "            # Set weights\n",
    "            if m == 0:\n",
    "                #TODO\n",
    "                 \n",
    "            else:\n",
    "                #TODO\n",
    "               \n",
    "            \n",
    "            # 1. Fit weak classifier and predict labels(using predict method) and Save it to list of weak classifiers.\n",
    "            #TODO\n",
    "\n",
    "\n",
    "            # 2. Calculate error of this weak classifier and save it to list of trainingterrors.\n",
    "            #TODO\n",
    "\n",
    "\n",
    "            # 3. Calculate alpha of this weak classifier and save it to list of alphas.\n",
    "            #TODO\n",
    "\n",
    "\n",
    "\n",
    "    def predict(self, X):\n",
    "\n",
    "        weak_preds = pd.DataFrame(index = range(len(X)), columns = range(self.M)) \n",
    "\n",
    "        for m in range(self.M):\n",
    "            y_pred_m = self.G_M[m].predict(X) * self.alphas[m]\n",
    "            weak_preds.iloc[:,m] = y_pred_m\n",
    "\n",
    "        y_pred = (1 * np.sign(weak_preds.T.sum())).astype(int)\n",
    "\n",
    "        return y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f27f8987",
   "metadata": {},
   "source": [
    "### Training and Evaluation (20 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2986c7a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO : Fit model\n",
    "\n",
    "\n",
    "#TODO : Predict on test data\n",
    "\n",
    "\n",
    "#TODO : Print evaluation metrics (Accurcy, Precission, Recall and f-score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39bb9216",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO : Do Scikit-Learn implementation of AdaBoost and print evaluation metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8fdcb6d",
   "metadata": {},
   "source": [
    "### Early Stopping (15 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac86c294",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO : Calculate validation error for different number of estimators "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6e06fea",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO : Plot validation error versus number of estimators figure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdfac096",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO : Print the best number for estimators and minimum value for validation error."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec46a972",
   "metadata": {},
   "source": [
    "### Weighted Error (10 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7471664f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO : Plot weighted training error versus number of estimators figure using training_errors attribute in Adaboost class."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22f513e8",
   "metadata": {},
   "source": [
    "### Question : Why does the weighted error tend to increase as the number of estimators increase? (5points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef5055b8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
